Image processing apparatus, method of processing image, and computer-readable recording medium

ABSTRACT

An image processing apparatus includes: an area extracting unit that extracts a candidate area of a classification target area in which a pixel value does not correspond to a three-dimensional shape of an imaging target based on pixel values of an intraluminal image acquired by imaging the inside of a lumen or information of a change in pixel values of peripheral pixels; and an area classifying unit that classifies the classification target area out of the candidate area based on the pixel values of the inside of the candidate area, a boundary portion of the candidate area, or a periphery portion of the candidate area.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2010-153355, filed on Jul. 5, 2010, theentire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, a methodof processing an image, and a computer-readable recording medium forprocessing an intraluminal image acquired by imaging the inside of alumen.

2. Description of the Related Art

In Japanese Laid-open Patent Publication No. 2007-236629, a method isdisclosed in which the three-dimensional image of a body tissue isestimated based on an intraluminal image (endoscopic image), and alesion site having a protruded shape such as a polyp is detected basedon shape feature data such as curvature at each position of theestimated three-dimensional shape. Here, in order to estimate thethree-dimensional shape, while pixel values corresponding to thethree-dimensional shape of the imaging target need to be acquired fromthe intraluminal image, in Japanese Laid-open Patent Publication No.2007-236629 described above, luminance information of the intraluminalimage is regarded as pixel values corresponding to the three-dimensionalimage, and the three-dimensional shape is estimated by performing ageometric conversion process based on the pixel values.

SUMMARY OF THE INVENTION

An image processing apparatus according to an aspect of the presentinvention includes an area extracting unit that extracts a candidatearea of a classification target area in which a pixel value does notcorrespond to a three-dimensional shape of an imaging target based onpixel values of an intraluminal image acquired by imaging the inside ofa lumen or information of a change in pixel values of peripheral pixels;and an area classifying unit that classifies the classification targetarea out of the candidate area based on the pixel values of the insideof the candidate area, a boundary portion of the candidate area, or aperiphery portion of the candidate area.

A method of processing an image according to another aspect of thepresent invention includes extracting a candidate area of aclassification target area in which a pixel value does not correspond toa three-dimensional shape of an imaging target based on pixel values ofan intraluminal image acquired by imaging the inside of a lumen orinformation of a change in pixel values of peripheral pixels; andclassifying the classification target area out of the candidate areabased on the pixel values of the inside of the candidate area, aboundary portion of the candidate area, or a periphery portion of thecandidate area.

An image processing program according to the present invention causes acomputer to perform: extracting a candidate area of a classificationtarget area in which a pixel value does not correspond to athree-dimensional shape of an imaging target based on pixel values of anintraluminal image acquired by imaging the inside of a lumen orinformation of a change in pixel values of peripheral pixels; andclassifying the classification target area out of the candidate areabased on the pixel values of the inside of the candidate area, aboundary portion of the candidate area, or a periphery portion of thecandidate area.

The above and other features, advantages and technical and industrialsignificance of this invention will be better understood by reading thefollowing detailed description of presently preferred embodiments of theinvention, when considered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an intraluminal image that is imaged byan endoscope;

FIG. 2 is a schematic diagram of three-dimensional pixel valueinformation of the intraluminal image illustrated in FIG. 1;

FIG. 3 is a schematic block diagram illustrating the main configurationof an image processing apparatus according to a first embodiment;

FIG. 4 is the entire flowchart illustrating the processing sequenceperformed by an image processing apparatus according to the firstembodiment;

FIG. 5 is a diagram illustrating the cross-section directions of a bloodvessel, a mirror reflection portion, a color-changed lesion, and amucous membrane undulation;

FIG. 6A is a schematic diagram illustrating a change in the pixel valuewithin the area of the mirror reflection portion along cross-sectiondirection a illustrated in FIG. 5;

FIG. 6B is a schematic diagram illustrating a change in the pixel valuewithin the area of a white lesion along cross-section direction billustrated in FIG. 5;

FIG. 6C is a schematic diagram illustrating a change in the pixel valuewithin the area of a blood vessel along cross-section direction cillustrated in FIG. 5;

FIG. 6D is a schematic diagram illustrating a change in the pixel valuewithin the area of a red or black lesion along cross-section direction dillustrated in FIG. 5;

FIG. 6E is a schematic diagram illustrating a change in the pixel valuewithin the area of a difference in level of the mucous membrane alongcross-section direction e illustrated in FIG. 5;

FIG. 6F is a schematic diagram illustrating a change in the pixel valuewithin the area of a mucous membrane groove along cross-sectiondirection f illustrated in FIG. 5;

FIG. 7 is a flowchart illustrating the detailed processing sequence ofan area classifying process according to a first embodiment;

FIG. 8 is a conceptual diagram illustrating pixel value complementingusing a morphology opening process;

FIG. 9 is a conceptual diagram illustrating pixel value complementingusing a morphology closing process;

FIG. 10 is a flowchart illustrating the detailed processing sequence ofa pixel value complementing process according to the first embodiment;

FIG. 11 is a schematic block diagram illustrating the main configurationof an image processing apparatus according to a second embodiment;

FIG. 12 is the whole flowchart illustrating the processing sequenceperformed by an image processing apparatus according to the secondembodiment;

FIG. 13 is a conceptual diagram illustrating the calculation of adifference value for each direction;

FIG. 14 is a diagram illustrating the cross-section directions of ablood vessel, a mirror reflection portion, a color-changed lesion, and amucous membrane undulation;

FIG. 15A is a schematic diagram illustrating a change in the pixel valuein the area boundary portion of the mirror reflection portion alongcross-section direction a illustrated in FIG. 14;

FIG. 15B is a schematic diagram illustrating a change in the pixel valuein the area boundary portion of the blood vessel along cross-sectiondirection b illustrated in FIG. 14;

FIG. 15C is a schematic diagram illustrating a change in the pixel valuein the area boundary portion of the color-changed lesion along thecross-section direction c illustrated in FIG. 14;

FIG. 15D is a schematic diagram illustrating a change in the pixel valuein the area boundary portion of a shape-changed lesion alongcross-section direction d illustrated in FIG. 14;

FIG. 15E is a schematic diagram illustrating a change in the pixel valuein the area boundary portion of a difference in level of the mucousmembrane along cross-section direction e illustrated in FIG. 14;

FIG. 15F is a schematic diagram illustrating a change in the pixel valuein the area boundary portion of a mucous membrane groove alongcross-section direction f illustrated in FIG. 14;

FIG. 16 is a flowchart illustrating the detailed processing sequence ofan area classifying process according to a second embodiment;

FIG. 17 is a conceptual diagram illustrating pixel value complementingusing function approximation;

FIG. 18 is a flowchart illustrating the detailed processing sequence ofa pixel value complementing process according to the second embodiment;

FIG. 19 is a schematic block diagram illustrating the main configurationof an image processing apparatus according to a third embodiment;

FIG. 20 is the whole flowchart illustrating the processing sequenceperformed by an image processing apparatus according to the thirdembodiment;

FIG. 21A is a schematic diagram illustrating a change in the pixel valuein the area periphery portion of the mirror reflection portion alongcross-section direction a illustrated in FIG. 5;

FIG. 21B is a schematic diagram illustrating a change in the pixel valuein the area periphery portion of the white lesion along cross-sectiondirection b illustrated in FIG. 5;

FIG. 21C is a schematic diagram illustrating a change in the pixel valuein the area periphery portion of the blood vessel along thecross-section direction c illustrated in FIG. 5;

FIG. 21D is a schematic diagram illustrating a change in the pixel valuein the area periphery portion of the red or black lesion alongcross-section direction d illustrated in FIG. 5;

FIG. 21E is a schematic diagram illustrating a change in the pixel valuein the area periphery portion of the difference in level of the mucousmembrane along cross-section direction e illustrated in FIG. 5;

FIG. 21F is a schematic diagram illustrating a change in the pixel valuein the area periphery portion of the mucous membrane groove alongcross-section direction f illustrated in FIG. 5;

FIG. 22 is a flowchart illustrating the detailed processing sequence ofan area classifying process according to the third embodiment;

FIG. 23 is a system configuration diagram illustrating the configurationof a computer system according to an embodiment of the presentinvention; and

FIG. 24 is a block diagram illustrating the configuration of a main unitconfiguring the computer system illustrated in FIG. 23.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings. In thisembodiment, an image processing apparatus that processes an intraluminalimage (image of the inside of a gastrointestinal tract) that is imagedby an endoscope will be described. The endoscope is a medical devicethat is used for observing the inside of a lumen such as agastrointestinal tract. The endoscope is a system device that isconfigured by: an insertion unit that has an illumination system, anoptical system, an imaging system, and the like for imaging an objectbeing built in the distal end thereof and is inserted into the inside ofa lumen, a casing unit that is connected to the insertion unit and has alight source, an image processing apparatus, and the like being builttherein; a display unit that displays the imaged intraluminal image; andthe like. However, the present invention is not limited to thisembodiment. Like reference numerals in the drawings denote likeelements.

FIG. 1 is a schematic diagram of an intraluminal image that is imaged byan endoscope. FIG. 2 is a schematic diagram of three-dimensional pixelvalue information of the intraluminal image illustrated in FIG. 1. Asillustrated in FIG. 1, basically, a mucous membrane 1 of an inner wallof a gastrointestinal tract is reflected on the intraluminal image, anda blood vessel 2 located under the surface layer of the mucous membrane,a mirror reflection portion 3, a lesion 4, and the like are alsoreflected on the intraluminal image at times. As such lesions 4, thereare a color-changed lesion 5 such as redness in which only the colorchanges with little change in the surface of the mucous membrane and ashape-changed lesion 6 such as a polyp in which there is a change in theshape of the surface of the mucous membrane. In addition, on theintraluminal image, a mucous membrane undulation 7 called a differencein level of the mucous membrane that is generated due to folding,winding, or the like of the structure of the mucous membrane or a mucousmembrane groove is reflected, and the inside of the lumen (a mucousmembrane located at a position far from the imaging system) is reflectedas well as a dark portion area 8. Here, for the shape-changed lesion 6,the mucous membrane undulation 7 such as the difference in level of themucous membrane or the mucous membrane groove, in addition to the mucousmembrane 1 and the dark portion area 8, the change in the pixel valuecorresponds to a three-dimensional shape. In contrast to this, the bloodvessel 2 located under the surface layer of the mucous membrane, themirror reflection portion 3, and the color-changed lesion 5 are portionswhich are influenced much by light absorption, mirror reflection, or thelike, and for which the change in the pixel value does not correspond toa three-dimensional shape. Specifically, as described above, theluminance value at the blood vessel 2 decreases due to absorption ofillumination light in blood. In addition, similarly, the luminance valueat the color-changed lesion 5 decreases as well due to absorption oflight in the blood. On the other hand, the luminance value at the mirrorreflection portion 3 increases regardless of the three-dimensional shapeof a body tissue. In addition, generally, the intraluminal image that isimaged by an endoscope is a color image that has pixel values forwavelength components of R (red), G (green), and B (blue) at each pixelposition.

The image processing apparatus according to this embodiment calculatespixel values (hereinafter, referred to as “three-dimensional pixel valueinformation”) corresponding to the three-dimensional shape of the mucousmembrane as a body tissue from an intraluminal image by processing theintraluminal image described as above. First, areas of the blood vessel2, the mirror reflection portion 3, and the color-changed lesion 5 forwhich the pixel values do not correspond to a three-dimensional shape ofthe imaging target (body tissue) are classified. Then, the classifiedareas of the blood vessel 2, the mirror reflection portion 3, and thecolor-changed lesion 5 are complemented based on the peripheral pixelvalues thereof, whereby the three-dimensional pixel values arecalculated. In addition, for the areas of any other mucous membrane 1,the shape-changed lesion 6, the mucous membrane undulation 7, and thedark portion area 8, three-dimensional pixel values are calculated basedon the pixel values, whereby the three-dimensional pixel valueinformation as illustrated in FIG. 2 is calculated.

First Embodiment

First, the configuration of an image processing apparatus 10 accordingto the first embodiment will be described. FIG. 3 is a schematic blockdiagram illustrating the main configuration of the image processingapparatus 10 according to the first embodiment. As illustrated in FIG.3, the image processing apparatus 10 according to the first embodimentincludes a calculation unit 20 and a recording unit 30. This imageprocessing apparatus 10, for example, is built in the above-describedendoscope and outputs calculated three-dimensional pixel valueinformation by receiving an intraluminal image that is imaged by theendoscope as an input and processing the intraluminal image.

The calculation unit 20 is realized by hardware such as a CPU andperforms various calculation processes used for calculating thethree-dimensional pixel value information based on the intraluminalimage. This calculation unit 20 includes an area extracting unit 21, anarea classifying unit 22, a pixel-value calculating unit 23, and apixel-value complementing unit 24.

The area extracting unit 21 includes a specific-color area extractingunit 211. This area extracting unit 21 is a functional unit that is usedfor extracting candidate areas (classification-target candidate area) ofclassification target areas from an intraluminal image. In the firstembodiment, the specific-color area extracting unit 211 extracts areasof specific colors as classification-target candidate areas based on thepixel values.

The area classifying unit 22 classifies areas of a blood vessel, amirror reflection portion, and a color-changed lesion out of theclassification-target candidate areas as classification target areasbased on the unevenness information and the smoothness of the curvedshape of each classification-target candidate area that is representedby the pixel values within the classification-target candidate area.This area classifying unit 22 includes a second derivative calculatingunit 221, an internal unevenness determining unit 222, and an internalsmoothness determining unit 223. The second derivative calculating unit221 calculates a second derivative value by applying a second derivativefilter to the pixels located within the classification-target candidatearea. The internal unevenness determining unit 222 determines theunevenness of the curved shape (change in the pixel value) of theclassification-target candidate area based on the sign of the secondderivative value of the pixels within the classification-targetcandidate area. The internal smoothness determining unit 223 determinesthe smoothness of the curved shape (change in the pixel value) of theclassification-target candidate area based on the absolute value of thesecond derivative value of the pixels within the classification-targetcandidate area.

The pixel-value calculating unit 23 calculates pixel values of aspecific wavelength component that is specified in accordance with thedegree of absorption or scattering inside a body tissue asthree-dimensional pixel values for an area other than the classificationtarget areas. This pixel-value calculating unit 23 includes anR-component selecting unit 231 that selects pixel values of an Rcomponent from an area other than the classification target areas withthe R component set as the specific wavelength component.

The pixel-value complementing unit 24 complements the three-dimensionalpixel values of the areas of the blood vessel, the mirror reflectionportion, and the color-changed lesion as the classification target areasbased on the pixel value of the specific-wavelength component of thearea other than the classification target areas. This pixel-valuecomplementing unit 24 includes a morphology processing unit 241 thatperforms a morphology process based on the pixel value of thespecific-wavelength component of the periphery of the classificationtarget area. The morphology processing unit 241 includes a structureelement generating unit 242 that generates a structure element used inthe morphology process based on feature data of the classificationtarget area.

The recording unit 30 is realized by: various memories such as a ROM asa flash memory for which update recording can be performed or the likeand a RAM; a hard disk that is built in or is connected through acommunication terminal; various types of recording medium such as aCD-ROM; a reading device thereof; and the like. In this recording unit30, a program used for realizing various functions included in the imageprocessing apparatus 10 by operating the image processing apparatus 10,data used during the execution of the program, and the like arerecorded. For example, in the recording unit 30, image data of anintraluminal image that is imaged by the endoscope and is input to theimage processing apparatus 10 is recorded. In addition, in the recordingunit 30, an image processing program 31 that is used for calculating thethree-dimensional pixel value information base on the intraluminal imageis recorded.

Next, the detailed processing sequence performed by the image processingapparatus 10 according to the first embodiment will be described. FIG. 4is an entire flowchart illustrating the processing sequence performed bythe image processing apparatus 10 according to the first embodiment. Theprocess described here is realized by executing the image processingprogram 31 recorded in the recording unit 30 by using the calculationunit 20.

As illustrated in FIG. 4, first, the calculation unit 20 acquires anintraluminal image as a processing target (Step a1). Through thisprocess, the intraluminal image that is imaged, for example, by anendoscope and is recorded in the recording unit 30 is read out so as tobe acquired.

Next, the specific-color area extracting unit 211 of the area extractingunit 21 extracts areas of specific colors from the intraluminal image asthe processing target as classification-target candidate areas based onthe pixel values (Step a3). Each area of the blood vessel, the mirrorreflection portion, or the color-changed lesion represents a specificcolor. For example, the blood vessel represents red or red-violet, themirror reflection portion represents white, and the color-changed lesionrepresents red, black, white, or the like. According to the firstembodiment, by extracting areas of specific colors,classification-target candidate areas as candidates for a blood vessel,a mirror reflection portion, or a color-changed lesion are extracted.

As methods of extracting specific-color areas, there are variousmethods. Here, a method is used in which a distribution of previoussample data is approximated as a probability model, and thespecific-color areas are extracted based on determinations using theprobability model. Specifically, a plurality of pixels belonging to eachcategory of the blood vessel, the mirror reflection, or thecolor-changed lesion is sampled in advance, and color feature data ofthe pixels is acquired. Here, the color feature data is a pixel value ofeach component of R, G or B, a value that is secondarily calculatedthrough a known conversion process based on the pixel values, a colordifference (YCbCr conversion), hue, saturation (HSI conversion), a colorratio (G/R or B/G), or the like. Next, a feature vectorFn=(fn_(—1, fn)_2, . . . , fn_k)^(t) that is formed from color featuredata of the sampled pixel is calculated. Here, fn_j denotes the j-thcolor feature data of the n-th sampled pixel, and k denotes the numberof the color feature data values. Then, an average vector p and avariance-covariance matrix Z are acquired based on this feature vectorFn by using the following Equation (1) and are recorded in the recordingunit 30. In the equation, ND is the number of sampled data values.

$\begin{matrix}{{\mu = {\frac{1}{ND}{\sum\limits_{n = 1}^{ND}{Fn}}}},{Z = {\frac{1}{ND}{\sum\limits_{n = 1}^{ND}{( {{Fn} - \mu} )( {{Fn} - \mu} )^{t}}}}}} & (1)\end{matrix}$

Actually, in Step a3, first, a determination index P(x) that indicateswhether or not each pixel is a specific color represented by eachcategory area is calculated for each category by using the followingEquation (2) based on the feature data x=(x_1, x_2, . . . , x_k)^(t)that is formed from the color feature data of each pixel of theintraluminal image as the processing target and the average vector p andthe variance-covariance matrix Z of each category that are acquired andrecorded in the recording unit 30 in advance. Here, |Z| is a matrixequation of Z, and Z⁻¹ is an inverse matrix of Z.

$\begin{matrix}{{P(x)} = {\frac{1}{( {2\pi} )^{k/2} \times {Z}^{1/2}}\exp \{ {( {x - \mu} )^{t} \times {- \frac{1}{2}}Z^{- 1} \times ( {x - \mu} )} \}}} & (2)\end{matrix}$

Thereafter, to a pixel for which the maximum value of the determinationindex P(x) is a predetermined threshold value or more, an integer value(label value) corresponding to a category representing the maximum valueis set, and, to a pixel for which the maximum value of the determinationindex P(x) is the threshold value or less, “0” is set, therebygenerating a candidate area image.

As above, in this example, although an extraction method using theprobability model is illustrated, the method is not particularlylimited, as long as the areas of specific colors corresponding to theareas of the blood vessel, the mirror reflection portion, thecolor-changed lesion can be extracted by using it. As a method otherthan that described in this example, a method may be used in which acolor feature vector, for example, representing a specific color is set,and an area of the specific color inside the intraluminal image isextracted based on a distance between the set vector and a featurevector data formed from color feature data of a pixel as a determinationtarget inside a feature space (a space in which a feature axisrepresenting the size relationship of the feature data extends).Alternatively, a method may be used in which a range representing aspecific color is set in a feature space, and an area of the specificcolor inside the intraluminal image is extracted based on the positionalrelationship between the set range and a feature vector that is formedfrom color feature data of a pixel as a determination target.Furthermore, the method is not limited to a method in which the area ofthe specific color is extracted by using the color feature data in unitsof a pixel is not limited, and a method may be used in which the insideof an image is divided into small areas, and then the area of thespecific color is extracted by using the color feature data in units ofa small area.

Next, the calculation unit 20 determines whether or not there is an areaextracted as an area of the specific color (a classification-targetcandidate area). This can be determined by determining whether or notthere is a pixel having a positive value as the pixel value in thecandidate area image. In a case where there is no extracted area (No inStep a5), the process proceeds to Step a15 to be described later.

On the other hand, in a case where there is an extracted area(classification-target candidate area) (Yes in Step a5), the areaclassifying unit 22 classifies the areas of the blood vessel, the mirrorreflection portion, and the color-changed lesion as the classificationtarget areas out of the classification-target candidate area based onthe unevenness information and the smoothness of the curved shapes ofthe classification-target candidate areas by performing an areaclassifying process (Step a7). There is a possibility that a differencein level of the mucous membrane, a mucous membrane groove, or the likeother than the blood vessel, the mirror reflection portion, and thecolor-changed lesion is included in the area of the specific color thatis extracted based on the color feature data in Step a3. The reason forthis is that this may represent a color that is close to red or black attimes. Thus, in the area classifying process according to the firstembodiment, both areas are classified based on the unevennessinformation and the smoothness of the curved shape of theclassification-target candidate area that is represented by pixel valuesof the inside of the classification-target candidate area.

Before the classification, pixels having the same pixel value within thecandidate area image need to be connected to each other so as to berecognized as one area. This can be realized by performing a knownlabeling process (reference: “Digital Image Processing”, CG-ARTSSociety, 181P, Labeling).

Here, before the detailed processing sequence of the area classifyingprocess is described, the features of the inside of the area of eachcategory area of the blood vessel, the mirror reflection portion, thecolor-changed lesion, the difference in level of the mucous membrane,and the mucous membrane groove included in the area of the specificcolor will be described. FIG. 5 is a diagram illustrating cross-sectiondirections a to f of category areas of a blood vessel 2, a mirrorreflection portion 3, a color-changed lesion 5, and a mucous membraneundulation 7. FIG. 6A is a schematic diagram illustrating a change inthe pixel value within the area of the mirror reflection portion 3 alongcross-section direction a illustrated in FIG. 5. FIG. 6B is a schematicdiagram illustrating a change in the pixel value within the area of awhite lesion as the color-changed lesion 5 illustrated in FIG. 5 alongcross-section direction b. FIG. 6C is a schematic diagram illustrating achange in the pixel value within the area of the blood vessel 2 alongcross-section direction c illustrated in FIG. 5. FIG. 6D is a schematicdiagram illustrating a change in the pixel value within the area alongcross-section direction d in a case where the color-change lesion 5illustrated in FIG. 5 is a red or black lesion. FIG. 6E is a schematicdiagram illustrating a change in the pixel value within the area of adifference in level of the mucous membrane as the mucous membraneundulation 7 illustrated in FIG. 5 along cross-section direction e. FIG.6F is a schematic diagram illustrating a change in the pixel valuewithin the area of a mucous membrane groove as the mucous membraneundulation 7 illustrated in FIG. 5 along cross-section direction f.

As described above, the pixel values of the intraluminal image arecomposed of values of the components R, G, and B. Although there aredifferences in the scale between the values, the tendencies of thechanges are similar to each other. Here, both change in the pixel valueof the mirror reflection portion 3 along cross-section direction aillustrated in FIG. 5 and the change in the pixel value of the whitelesion (5) along cross-section direction b, as illustrated in FIGS. 6Aand 6B, represent convex shapes in which the pixel value of the insideof the area is greater than that of the periphery. On the other hand,the change in the pixel value of the blood vessel 2 along cross-sectionc, the change in the pixel value of the red or black lesion (5) alongcross-section direction d, and the change in the pixel value of thedifference in level (7) of the mucous membrane along cross-sectiondirection e illustrated in FIG. 5, as illustrated in FIGS. 6C, 6D, 6E,and 6F, represent concave shapes in which the pixel value is less thanthat of the periphery. In addition, since the change in the pixel valueof the blood vessel 2 or the red or black lesion (5) is an integraltarget, as illustrated in FIGS. 6C and 6D, the pixel value of the insideof the area smoothly change. In contrast to this, since the differencein level (7) of the mucous membrane along cross-section direction eillustrated in FIG. 5 includes a boundary of another mucous membraneinside the area, the change in the pixel value, as illustrated by beingsurrounded by a broken line in FIG. 6E, includes a portion E11 in whichthe pixel value does not smoothly change. Similarly, the mucous membranegroove (7) along cross-section direction f illustrated in FIG. 5, asillustrated by being surrounded by a broken line in FIG. 6F, includes aportion E12 in which the pixel value does not smoothly change. In thearea classifying process according to the first embodiment classifiesthe areas of the blood vessel, the mirror reflection portion, and thecolor-changed lesion as classification target areas out of theclassification-target candidate areas based on such features.

In the actual area classifying process, the classification-targetcandidate areas are sequentially set as processing targets, and it issequentially determined whether or not each classification-targetcandidate area is a classification-target area. FIG. 7 is a flowchartillustrating the detailed processing sequence of an area classifyingprocess according to the first embodiment. As illustrated in FIG. 7,first, the area classifying unit 22 sets classification-target candidateareas (hereinafter, the classification-target candidate areas as theprocessing targets are referred to as “processing candidate areas”) asprocessing targets (Step b1).

Next, the second derivative calculating unit 221 calculates a secondderivative value at each pixel by applying a known second derivativefilter (reference: “Digital Image Processing”, CG-ARTS Society, 118P,Second derivative Filter) to the pixel value of any one of thecomponents R, G, and B of the pixel located within a processingcandidate area in Step b3. At this time, in order to reduce theinfluence of a noise, for example, a Laplacian of Gaussian (LOG) filterthat is a second derivative filter accompanying smoothing is used.

Next, the internal unevenness determining unit 222 determines theunevenness of the curved shape (change in the pixel value) of theprocessing candidate area based on the pixel values within theprocessing candidate area (Step b5). The second derivative filterbasically has a structure in which a difference between the averagevalue of the peripheral pixels and the value of the center pixel or theaverage value of pixels located near the center pixels is acquired, andaccordingly, the sign thereof is inverted in accordance with theunevenness of the change in the pixel value. Thus, the determination ofthe unevenness of the curved shape is performed by determining theunevenness of the change in the pixel value based on the sign of theaverage value of the second derivative values of the pixels within theprocessing candidate area. In a case where the result of the unevennessdetermination is a convexity, the process proceeds to Step b7, and thearea classifying unit 22 classifies the processing candidate area as themirror reflection portion or the white color lesion. Alternatively, thedetermination of the unevenness of the curved shape may be performed bycomparing the numbers of the positive signs and the negative signs ofthe second derivative values of the pixels within the processingcandidate area. Furthermore, the unevenness determination may beperformed based on the sign of a maximum value of the second derivativevalues.

On the other hand, in a case where the result of the unevennessdetermination is a concavity, the process proceeds to Step b9, and theinternal smoothness determining unit 223 determines the smoothness ofthe curved shape (change in the pixel value) of the processing candidatearea based on the pixel values within the processing candidate area. Fordetermining the smoothness, a second derivative value is used as well.In a case where the change in the pixel value is smooth, a differencebetween values of the peripheral pixel and the center pixel is small,and accordingly, the absolute value of the second derivative value issmall. Accordingly, the smoothness is determined by determining whetheror not the average value of the absolute values of the second derivativevalues within the processing candidate area is a threshold value set inadvance or less. The determining of the smoothness is not limited tobeing based on the average value of the absolute values of the secondderivative values within the processing candidate area. Thus, it may beconfigured such that a maximum value of the second derivative valueswithin the processing candidate area is calculated, and threshold valueprocessing is performed for the maximum value.

In a case where the change in the pixel value within the processingcandidate area is determined to be smooth in Step b9, the processproceeds to Step b11, and the area classifying unit 22 classifies theprocessing candidate area as a blood vessel or a red or black lesion. Onthe other hand, in a case where the change is determined not to besmooth in Step b9, the process proceeds to Step b13, and the areaclassifying unit 22 classifies the processing candidate area as adifference in level of a mucous membrane or a mucous membrane groove.

Thereafter, the area classifying unit 22 determines whether all theclassification-target candidate areas have been classified. In a casewhere all the classification-target candidate areas have not beenclassified (No in Step b15), the area classifying unit 22 sets aclassification-target candidate area, which has not been classified, asthe processing candidate area (Step b17), and performs the process ofSteps b3 to b15 again. On the other hand, in a case where all theclassification-target candidate areas have been classified (Yes in Stepb15), the process is returned to Step a7 illustrated in FIG. 4 andproceeds to Step a9. Consequently, a classification-target area image isgenerated in which positive values (label values) used for areaidentification are set to the pixels located in the areas classified asa blood vessel, a mirror reflection portion, or a color-changed lesionthat is the classification target area, and “0” is set to pixels locatedin an area other than the classification target area.

Next, the calculation unit 20 determines whether there is an areaclassified as a blood vessel, a mirror reflection portion, or acolor-changed lesion that is the classification target area. This can bedetermined by determining whether or not there is a pixel having apositive value as the pixel value in the classification-target areaimage. In a case where there is no area classified as a blood vessel, amirror reflection portion, or a color-changed lesion (No in Step a9),the process proceeds to Step a15 to be described later.

On the other hand, there is an area (classification target area)classified as a blood vessel, a mirror reflection portion, or acolor-change lesion (Yes in Step a9), the R-component selecting unit 231of the pixel-value calculating unit 23 selects the pixel value of the Rcomponent in an area other than the classification target area, andsubsequently, the pixel-value calculating unit 23 sets the pixel valueof the selected R component as a three-dimensional pixel value in anarea other than the classification target area (Step a11). Here, sincethe R component is a wavelength component located far from theabsorption band of the blood and is a long-wavelength component, it isdifficult for the R component to be influenced by absorption orscattering occurring inside a body, and the R component represents apixel value corresponding to a three-dimensional shape of the bodyorganization. Thus, according to the first embodiment, a specificwavelength component that is specified in accordance with the degree ofabsorption or scattering inside the body is set as the R component.

In this example, a method is illustrated in which the R component isselected from the intraluminal image of components R, G, and B. However,in a case where an intraluminal image of components cyan (C), magenta(M), and yellow (Y) that are complementary colors of R, G, and B ishandled, an R component may be selected after the components C, M, and Yare converted into components R, G, and B.

Next, the pixel-value complementing unit 24 performs a pixel-valuecomplementing process, thereby complementing the pixel values in theareas of a blood vessel, a mirror reflection portion, and acolor-changed lesion that are the classification target areas (Stepa13). In the first embodiment, the pixel-value complementing process isperformed by using a morphology process. Through the process performedhere, a three-dimensional pixel value in the classification target areais complemented based on the pixel value of the specific wavelengthcomponent calculated as a three-dimensional pixel value of an area (tobe specific, a periphery area of the classification target area) otherthan the classification target area in the previous stage of Step a11.

FIG. 8 is a conceptual diagram illustrating pixel value complementingusing a morphology opening process, and FIG. 9 is a conceptual diagramillustrating pixel value complementing using a morphology closingprocess. The morphology opening process illustrated in FIG. 8 is aprocess of acquiring a locus through which a maximum value of the outerperiphery of a structure element passes when a reference diagram calledstructure element is moved while being circumscribed to a target imagefrom the side on which a pixel value of the target image is small in athree-dimensional space in which each pixel value is regarded aselevation. On the other hand, the morphology closing process illustratedin FIG. 9 is a process of acquiring a locus through which a minimumvalue of the outer periphery of the structure element passes when thestructure element is moved while being circumscribed to a target imagefrom the side on which a pixel value of the target image is large in thesame three-dimensional space. These are known techniques (reference:Obata Hidehumi, “Morphology”, Corona Publishing Co., Ltd.).

FIG. 10 is a flowchart illustrating the detailed processing sequence ofa pixel value complementing process according to the first embodiment.In the pixel-value complementing process, as illustrated in FIG. 10,first, the pixel-value complementing unit 24 sets a classificationtarget area (hereinafter, this classification target area as aprocessing target is referred to as a “complementation target area”) asa processing target (Step c1). Then, the pixel-value complementing unit24 calculates area feature data of the complementation target area (Stepc3). There are two sets of the area feature data described hereincluding the unevenness information of the change in the pixel value(curved shape) as a result of the unevenness determination performed inStep b5 illustrated in FIG. 7 and an area width that is newlycalculated. The area width can be calculated by solving simultaneousequations represented in the following Equation (3) in a case where anarea Area and a perimeter Peri (reference: “Digital Image Processing”,CG-ARTS Society, 182P, Area and Perimeter) as shape feature data of thecomplementation target area are calculated by using a known technique,and then the complementation target area is approximated as a rectanglehaving a length L and a width W.

$\begin{matrix}\{ \begin{matrix}{{Peri} = {2 \times ( {W + L} )}} \\{{Area} = {W \times L}}\end{matrix}  & (3)\end{matrix}$

For example, as illustrated in FIG. 8( a), the unevenness informationrepresenting the convexity and the area width R21 are used as areafeature data when the unevenness information is a convexity. Further, asillustrated in FIG. 9( a), the unevenness information representing theconcavity and the area width R22 are used as area feature data when theunevenness information is a concavity. Here, in Step a11 of the previousstage illustrated in FIG. 4, the pixel value of the R component in anarea other than the classification target area is selected, and thepixel value of the selected R component is set as a three-dimensionalpixel value in the area other than the classification target area. InFIGS. 8( a) and 9(a), on the area periphery, the three-dimensional pixelvalue is denoted by a solid line, and, for the inside of the area thatis complemented based on the three-dimensional pixel value of the areaperiphery, the pixel value of the intraluminal image is denoted by abroken line.

Next, the structure element generating unit 242, as illustrated in FIG.10, sets the morphology structure element (Step c5). Here, unlessstructure elements that are larger than the area widths R21 and R22illustrated in FIGS. 8( a) and 9(a) are used, the pixel values insidethe area cannot be complemented based on the three-dimensional pixelvalues of the area periphery. Accordingly, in Step c5 illustrated inFIG. 10, as illustrated in FIGS. 8( b) and 9(b), structure elements F21and F22 of sizes larger than the calculated area widths R21 and R22 areset.

Then, the morphology processing unit 241, as illustrated in FIG. 10,selects a morphology process (Step c7) and complements the pixel valuesby performing the selected morphology process in Step c9. In order toprevent the complementation process from being influenced by the changein the pixel value on the area boundary, the type of morphology processto be applied is changed based on whether the unevenness information isthe convexity or the concavity.

Actually, in a case where the complementation target area is a convexarea as illustrated in FIG. 8( a), the morphology opening process isselected, and the pixel values are complemented by applying the selectedmorphology opening process. In other words, as illustrated in FIG. 8(c), the structure element F21 illustrated in FIG. 8( b) is moved whilebeing circumscribed to the area from the lower side of the area (fromthe side on which the pixel value is small), and as illustrated in FIG.8( d), by complementing the pixel values of the inside of the area,three-dimensional pixel values inside the area are acquired. On theother hand, in a case where the complementation target area, asillustrated in FIG. 9( a), is a convex area, the morphology closingprocess is selected, and the pixel values area complemented by using theselected morphology closing process. In other words, as illustrated inFIG. 9( c), the structure element F22 illustrated in FIG. 9( b) is movedwhile being circumscribed to the area from the upper side of the area(from the side on which the pixel value is large), and as illustrated inFIG. 9( d), by complementing the pixel values of the inside of the area,the three-dimensional pixel values inside the area are acquired.

Thereafter, the pixel-value complementing unit 24, as illustrated inFIG. 10, determines whether the pixel values of all the complementationtarget area have been complemented. In a case where the pixel values ofall the complementation target areas have not been complemented (No inStep c11), the pixel-value complementing unit 24 sets a classificationtarget area that has not been complemented as a complementation targetarea (Step c13) and performs the process of Steps c3 to c1 again. On theother hand, in a case where the pixel values of all the classificationtarget areas have been complemented (Yes in Step c11), the process isreturned to Step a13 illustrated in FIG. 4 and then proceeds to Step a17thereafter.

In addition, in a case where it is determined that there is no extractedarea in Step a5 (No in Step a5), or in a case where it is determinedthat there is no area classified as an area of the blood vessel, themirror reflection portion, or the color-changed lesion in Step a9 (No inStep a9), the R-component selecting unit 231 of the pixel-valuecalculating unit 23 selects the pixel values of the R component of theentire area of the intraluminal image, and subsequently, the pixel-valuecalculating unit 23 sets the pixel values of the selected R component asthe three-dimensional pixel values of the pixels located in the entirearea of the intraluminal image (Step a15).

Then, finally, the calculation unit 20 outputs the three-dimensionalpixel value information, in which the pixel value of each pixel of theintraluminal image is set as the three-dimensional pixel valuecalculated in Steps a11 and a13 or the three-dimensional pixel valuecalculated in Step a15 (Step a17) and ends the process performed in theimage processing apparatus 10.

As described above, according to the first embodiment, areas of aspecific color, which represent the specific color, are extracted asareas that are strongly influenced by light absorption, mirrorreflection, or the like, more particularly, the classification-targetcandidate areas as candidates for the areas of a blood vessel, a mirrorreflection portion, and a color-changed lesion based on the pixel valuesof the intraluminal image. Then, based on the unevenness information ofthe curved shape and the smoothness of the curved shape of theclassification-target candidate area that is represented by the pixelvalues of the inside of the classification-target candidate area, areasof a blood vessel, a mirror reflection portion, and a color-changedlesion that are strongly influenced by light absorption, mirrorreflection, or the like out of the classification-target candidate areasare classified as the classification target areas. Accordingly, areas ofthe intraluminal image of which the pixel values do not correspond tothe three-dimensional shape as an imaging target can be specified.

In addition, according to the first embodiment, by acquiring pixelvalues of a specific wavelength component that is specified inaccordance with the degree of absorption or scattering inside the bodyare acquired for an area other than the classification target areas ofwhich the pixel values specified as above do not correspond to thethree-dimensional shape of the imaging target, the three-dimensionalpixel values corresponding to the three-dimensional shape of the bodytissue are acquired. Then, by complementing the pixel values of theareas of a blood vessel, a mirror reflection portion, and acolor-changed lesion as the classification target areas by performing amorphology process based on the pixel values of the specific wavelengthcomponent that are acquired, for example, as the three-dimensional pixelvalues of the periphery of the classification target area, thethree-dimensional pixel values of the inside of the classificationtarget area are acquired. Accordingly, the three-dimensional pixel valuethat appropriately represents the three-dimensional shape of the bodytissue in the entire area of the intraluminal image can be calculatedand be output as the three-dimensional pixel value information.

Second Embodiment

First, the configuration of an image processing apparatus according to asecond embodiment will be described. FIG. 11 is a schematic blockdiagram illustrating the main configuration of an image processingapparatus 10 a according to the second embodiment. The same referencenumeral is assigned to the same configuration as that described in thefirst embodiment. The image processing apparatus 10 a according to thesecond embodiment includes a calculation unit 20 a and a recording unit30 a, as illustrated in FIG. 11. This image processing apparatus 10 a,for example, is built in an endoscope and outputs calculatedthree-dimensional pixel value information by receiving an intraluminalimage that is imaged by the endoscope as an input and processing theintraluminal image.

The calculation unit 20 a includes an area extracting unit 21 a, an areaclassifying unit 22 a, a pixel-value calculating unit 23 a, and apixel-value complementing unit 24 a.

The area extracting unit 21 a is a functional unit used for extractingcandidate areas (classification-target candidate areas) forclassification target areas from an intraluminal image. According to thesecond embodiment, the area extracting unit 21 a extracts theclassification-target candidate areas based on information of the changein the pixel value with respect to periphery pixels. This areaextracting unit 21 a includes a concave area/convex area extracting unit212 a that extracts a concave area that represents a pixel value lessthan the average value of pixel values of the periphery pixels and aconvex area that represents a pixel value greater than the average valueof the pixel values of the periphery pixels. This concave area/convexarea extracting unit 212 a includes a directional difference calculatingunit 213 a, a maximum value/minimum value calculating unit 214 a, and athreshold value processing unit 215 a. The directional differencecalculating unit 213 a calculates a difference value between a pixel ofinterest and an average value of periphery pixels that oppose each otherin a predetermined direction with the pixel of interest located at thecenter thereof for each of a plurality of directions. The maximumvalue/minimum value calculating unit 214 a calculates a maximum valueand a minimum value of the difference values. The threshold valueprocessing unit 215 a performs threshold-value processing for themaximum value and the minimum value.

The area classifying unit 22 a classifies areas of a blood vessel, amirror reflection portion, and a color-changed lesion out of theclassification-target candidate areas as classification target areasbased on the steepness of the change in the pixel value of the boundaryportion of the classification-target candidate area. This areaclassifying unit 22 a includes a first derivative calculating unit 224 aand a boundary steepness determining unit 225 a. The first derivativecalculating unit 224 a calculates a first derivative value by applying afirst derivative filter to the pixel of the boundary portion of theclassification-target candidate area. The boundary steepness determiningunit 225 a determines the steepness of the change in the pixel value ofthe boundary portion of the classification-target candidate area basedon the first derivative value of the pixel of the boundary portion ofthe classification-target candidate area.

The pixel-value calculating unit 23 a calculates pixel values of aspecific wavelength component that is specified in accordance with thedegree of absorption or scattering inside a body tissue asthree-dimensional pixel values for an area other than the classificationtarget areas. This pixel-value calculating unit 23 a includes aspectroscopic information estimating unit 232 a and aspecific-wavelength component calculating unit 233 a. The spectroscopicinformation estimating unit 232 a estimates spectroscopic information ofa body tissue based on a plurality of wavelength components of pixelvalues of an area other than the classification target areas. Thespecific-wavelength component calculating unit 233 a calculates a pixelvalue of a specific wavelength component based on the estimatedspectroscopic information of the body tissue.

The pixel-value complementing unit 24 a complements thethree-dimensional pixel values of the areas of the blood vessel, themirror reflection portion, and the color-changed lesion as theclassification target areas based on the pixel values of thespecific-wavelength component of the area other than the classificationtarget areas. This pixel-value complementing unit 24 a includes afunction approximation unit 243 a that approximates the pixel values ofthe classification target area as a function based on the pixel valuesof the periphery of the classification target area. The functionapproximation unit 243 a includes a sample pixel selecting unit 244 athat selects sample pixels used for the function approximation from atleast two periphery areas opposing each other with the classificationtarget area interposed therebetween.

In the recording unit 30 a, an image processing program 31 a used forcalculating three-dimensional pixel value information from theintraluminal image is recorded.

Next, the detailed processing sequence performed by the image processingapparatus 10 a according to the second embodiment will be described.FIG. 12 is an entire flowchart illustrating the processing sequenceperformed by the image processing apparatus 10 a according to the secondembodiment. The process described here is realized by executing theimage processing program 31 a recorded in the recording unit 30 a byusing the calculation unit 20 a.

First, the calculation unit 20 a acquires an intraluminal image as aprocessing target (Step d1).

Next, the concave area/convex area extracting unit 212 a of the areaextracting unit 21 a extracts concave areas and convex areas from anintraluminal image as a processing target based on the pixel values ofthe component G as classification-target candidate areas (Step d3). Asillustrated in FIGS. 6A, 6B, 6C, and 6D referred to in the firstembodiment, for each area of a blood vessel, a mirror reflectionportion, or a color-changed lesion, the curved shape represented by thepixel values of the inside of the area is a convex shape or a concaveshape. Accordingly, the pixel values of the inside of the area aredifferent from the pixel values of the periphery pixels. Described inmore detail, the area of the mirror reflection portion or the whitelesion area is formed as a convex area having a pixel value greater thanthe average value of the pixel values of the periphery pixels. On theother hand, the area of a blood vessel or a red or black lesion area isformed as a concave area having a pixel value less than the averagevalue of the pixel values of the periphery pixels. Thus, according tothe second embodiment, by extracting the convex areas and the concaveareas (hereinafter, these will be collectively referred to as“concavo-convex areas”) of the pixel values from the intraluminal image,classification-target candidate areas as candidates for a blood vessel,a mirror reflection portion, and a color-changed lesion are extracted.

While there are various methods of extracting the concavo-convex areasof the pixel values, here, a method is illustrated in which theconcavo-convex areas are detected by calculating a difference betweenvalues of a pixel of interest and periphery pixels for each direction.FIG. 13 is a conceptual diagram illustrating the calculation of adifference value for each direction. In the second embodiment, asillustrated in FIG. 13, the difference values are calculated for fourdirections including the horizontal direction, the vertical direction, adirection inclined toward the upper right side (first incliningdirection) and a direction inclined toward the lower right side (secondinclining direction). Described in more detail, when a pixel located atthe center is set as the pixel P3 of interest, a difference value dH forthe horizontal direction is calculated based on the pixel value of thepixel P3 of interest and the pixel values of periphery pixels P311 andP312 that are apart from the pixel P3 of interest by a radius r of acircle denoted by a dashed-dotted line in FIG. 13 along the horizontaldirection. Similarly, a difference value dV for the vertical directionis calculated based on the pixel value of the pixel P3 of interest andthe pixel values of periphery pixels P321 and P322 that are apart fromthe pixel P3 of interest by the radius r in the vertical direction. Inaddition, a difference value dD1 for the first inclining direction iscalculated based on the pixel value of the pixel P3 of interest and thepixel values of periphery pixels P331 and P332 that are apart from thepixel P3 of interest by the radius r in the first inclining direction,and a difference value dD2 for the second inclining direction iscalculated based on the pixel value of the pixel P3 of interest and thepixel values of periphery pixels P341 and P342 that are apart from thepixel P3 of interest by the radius r in the second inclining direction.

Actually, first, the directional difference calculating unit 213 a, foreach pixel (x, y) of the intraluminal image, calculates the differencevalue dH between the pixel value of each pixel and an average value ofperiphery pixels that oppose each other in the horizontal direction, adifference value dV between the pixel value of each pixel and an averagevalue of periphery pixels opposing each other in the vertical direction,a difference value dD1 between the pixel value of each pixel and anaverage value of periphery pixels opposing each other in the firstinclining direction, and a difference value dD2 between the pixel valueof each pixel and an average value of periphery pixels opposing eachother in the second inclining direction, using the following Equations(4) to (7).

dH(x,y)=P(x,y)−0.5×(P(x−r,y)+P(x+r,y))  (4)

dV(x,y)=P(x,y)−0.5×(P(x,y−r)+P(x,y+r))  (5)

dD1(x,y)=P(x,y)−0.5×(P(x−r′,y+r′)+P(x+r′,y−r′))  (6)

dD2(x,y)=P(x,y)−0.5×(P(x−r′,y−r′)+P(x+r′,y+r′))  (7)

Here, r′ represented in Equations (6) and (7) is a constant acquired byrounding off r/(2^(0.5)).

Here, P(x, y) is a pixel value of the component G at the coordinates(x,y) of the intraluminal image. In addition, r corresponds to “r”illustrated in FIG. 13 and is a parameter representing the pixel rangeat the time of calculating the difference value. By decreasing the valueof r, a concavo-convex area having large concavo-convex patterns can bedetected. On the other hand, by increasing the value of r, aconcavo-convex area having small concavo-convex patterns can bedetected. The value of r may be statically set in advance or may be setdynamically based on an image or the like. Here, the reason for usingthe component G is that the sensitivity near the absorption band ofblood can be acquired, and a blood, a color-changed lesion, or the likecan be easily extracted as a concavo-convex area. The extraction of theconcavo-convex area is not limited to being performed by using thecomponent G and may be performed by using another color component, avalue that is secondarily calculated through a known conversion process,luminance, a color difference (YCbCr conversion), hue, saturation,intensity (HSI conversion), a color ratio, or the like.

Then, after the maximum value/minimum value calculating unit 214 acalculates a maximum value of the difference values dH, dV, dD1, and dD2for each direction, the threshold value processing unit 215 a extractsconvex areas by extracting pixels for which the calculated maximum valueis a predetermined threshold value or more. In addition, after themaximum value/minimum value calculating unit 214 a calculates a minimumvalue (negative value) of the difference values dH, dV, dD1, and dD2 foreach direction, the threshold value processing unit 215 a extractsconcave areas by extracting pixels for which the calculated minimumvalue is a predetermined threshold value or less. Consequently, forexample, a candidate area image is generated in which positive values(label values) used for identifying concavo-convex patterns are set tothe pixels of the extracted area, and “0” is set to the other pixels.

As above, although the extraction method using the difference value foreach direction is illustrated in this example, here, a method other thanthe method described in this example may be used as long asconcavo-convex areas can be extracted from an intraluminal image byusing it. For example, concavo-convex areas may be extracted from theintraluminal image based on a correlation value with respect to a modelshape of concavo-convex patterns.

Next, the calculation unit 20 a, as illustrated in FIG. 12, determineswhether or not there is an extracted area as the concavo-convex area(classification-target candidate area). This can be determined bydetermining whether or not there is a pixel of which the pixel value isa positive value in the candidate area image. In a case where there isno extracted area (No in Step d5), the process proceeds to Step d17 tobe described later.

On the other hand, in a case where there is an extracted area(classification-target candidate area) (Yes in Step d5), the areaclassifying unit 22 a classifies areas of a blood vessel, a mirrorreflection portion, and a color-changed lesion, which are theclassification target areas, out of the classification-target candidateareas based on the steepness of the change in the pixel value of theboundary portion of the classification-target candidate area byperforming an area classifying process (Step d7). In the concavo-convexarea extracted based on the pixel value change information with respectto the periphery pixel in Step d3, there is a possibility that an mucousmembrane undulation including a difference in level of the mucousmembrane, a mucous membrane groove, the shaped-changed lesion, or thelike other than the blood vessel, the mirror reflection portion, and thecolor-changed lesion is included. The reason for this is that when adistance between the surface of the mucous membrane and the imagingsystem changes due to the mucous membrane undulation or the shape changein the mucous membrane, the pixel value changes in accordance with thechange in the distance. Accordingly, in the area classifying processaccording to the second embodiment, both areas are classified based onthe steepness of the change in the pixel value of the boundary portionof the classification-target candidate area.

In addition, before the classification, pixels having the same pixelvalue within the candidate area image are connected to each otherthrough a labeling process, similarly to the first embodiment.

Here, before the detailed processing sequence of the area classifyingprocess is described, the features of the area boundary portion of eachcategory area that is included in the concavo-convex area will bedescribed. FIG. 14 is a diagram illustrating the cross-sectiondirections a to f of each category of a blood vessel 2, a mirrorreflection portion 3, a color-changed lesion 5, the shape-changed lesion6, and a mucous membrane undulation 7. FIG. 15A is a schematic diagramillustrating a change in the pixel value in the area boundary portion ofthe mirror reflection portion 3 along cross-section direction aillustrated in FIG. 14. FIG. 15B is a schematic diagram illustrating achange in the pixel value in the area boundary portion of the bloodvessel 2 along cross-section direction b illustrated in FIG. 14. FIG.15C is a schematic diagram illustrating a change in the pixel value inthe area boundary portion of the color-changed lesion (for example, ared lesion) 5 along the cross-section direction c illustrated in FIG.14. FIG. 15D is a schematic diagram illustrating a change in the pixelvalue in the area boundary portion of the shape-changed lesion 6 alongcross-section direction d illustrated in FIG. 14. FIG. 15E is aschematic diagram illustrating a change in the pixel value in the areaboundary portion along cross-section direction e in a case where themucous membrane undulation 7 illustrated in FIG. 14 is a difference inlevel of a mucous membrane. FIG. 15F is a schematic diagram illustratinga change in the pixel value in the area boundary portion alongcross-section direction f in a case where the mucous membrane undulation7 illustrated in FIG. 14 is a mucous membrane groove.

Here, in the mirror reflection portion 3 along cross-section direction aillustrated in FIG. 14, the blood vessel 2 along cross-section directionb, and the color-changed lesion 5 along the cross-section direction c,due to the change in the pixel value caused by local reflection or lightabsorption, the change in the pixel value in the boundary is steep asillustrated in FIGS. 15A, 15B, and 15C. On the other hand, in theshape-changed lesion 6 along cross-section direction d illustrated inFIG. 14 or the mucous membrane undulation 7 along cross-sectiondirections e and f, the change in the pixel value on the boundary otherthan a boundary that is caused by occlusion (an area that is not imagedby being shielded by a foreground when a three-dimensional object istwo-dimensionally imaged), as illustrated in the portions E41 to E44surrounded by broken lines in FIGS. 15D, 15E, and 15F, is continuous inthe shape, and accordingly, the change in the pixel is not steep. Such adifference is more prominent as the wavelength is closer to theabsorption band of the blood such as the component G and the componentB. In the area classifying process of the second embodiment, the areasof a blood vessel, a mirror reflection portion, and a color-changedlesion are classified out of the classification-target candidate areasbased on such features.

FIG. 16 is a flowchart illustrating the detailed processing sequence ofthe area classifying process according to the second embodiment. In thearea classifying process, as illustrated in FIG. 16, first, the areaclassifying unit 22 a sets a classification-target candidate area(processing candidate area) as a processing target (Step e1).

Next, the first derivative calculating unit 224 a calculates a firstderivative value of each pixel by applying a known first derivativefilter (reference: “Digital Image Processing”, CG-ARTS Society, 114P,Differential Filter) to the pixel values of the component G or B of theboundary portion of the candidate processing area in Step e3. At thistime, in order to reduce the influence of a noise, for example, a Sobelfilter that is a first derivative filter accompanying smoothing is used.In addition, in order to recognize boundary pixels of the processingcandidate area, a known contour tracking process (reference: “DigitalImage Processing”, CG-ARTS Society, 178P, Contour Tracking) may be used.

Next, the boundary steepness determining unit 225 a determines whetheror not the change in the pixel of the boundary portion of the processingcandidate processing area is steep. Actually, the absolute values of thefirst derivative values are acquired for every boundary pixel, theacquired first derivative values are sorted in the descending order, anda first derivative value corresponding to a half of the total number ofthe boundary pixels from the smaller side is selected, thereby acquiringan average value. The reason for this is for excluding the influence ofthe occlusion described above. Then, by determining whether or not theaverage value is a predetermined threshold value or more, it isdetermined whether or not the change in the pixel value of the boundaryportion of the processing candidate area is steep. In a case where thechange in the pixel value of the boundary portion of the processingcandidate area is determined not to be steep (No in Step e5), the areaclassifying unit 22 a classifies the processing candidate area as amucous membrane undulation or a shape-changed lesion (Step e7).

On the other hand, in a case where the change in the pixel value of theboundary portion of the processing candidate area is determined to besteep (Yes in Step e5), the area classifying unit 22 a classifies theprocessing candidate area as a blood vessel, a mirror reflectionportion, or a color-changed lesion (Step e9).

Thereafter, the area classifying unit 22 a determines whether all theclassification-target candidate areas have been classified. In a casewhere all the classification-target candidate areas have not beenclassified (No in Step e11), the area classifying unit 22 a sets aclassification-target candidate area that has not been classified as theprocessing candidate area (Step e13) and performs the process of Stepse3 to ell again. On the other hand, in a case where all theclassification-target candidate areas have been classified (Yes in Stepell), the process is returned to Step d7 illustrated in FIG. 12 andproceeds to Step d9. Consequently, similarly to the first embodiment, aclassification-target area image is generated in which positive values(label values) used for identifying areas are set to the pixels of theareas classified as the blood vessel, the mirror reflection portion, andthe color-changed lesion as the classification target areas, and “0” isset to pixels of the areas other than the classification target areas.

Next, the calculation unit 20 a determines whether or not there is anarea that is classified as a blood vessel, a mirror reflection portion,or a color-changed lesion that is the classification target area. Thiscan be determined by determining whether or not there is a pixel havinga positive value as its pixel value in the classification target areaimage. In a case where there is no area classified as the blood vessel,the mirror reflection portion, or the color-changed lesion (No in Stepd9), the process proceeds to Step d17 to be described later.

On the other hand, in a case where there is an area (classificationtarget area) classified as the blood vessel, the mirror reflectionportion, or the color-changed lesion (Yes in Step d9), first, thespectroscopic information estimating unit 232 a of the pixel-valuecalculating unit 23 a estimates, for example, spectroscopic reflectanceas spectroscopic information of an area other than the classificationtarget area based on the pixel values of components R, G, and B of areasother than the classification target area (Step d11). In addition, theestimated spectroscopic information is not limited to the spectroscopicreflectance but may be spectroscopic transmittance, spectroscopicabsorbance, or the like.

Described in more detail, the spectroscopic reflectance of each of aplurality of body tissues is collected in advance, and principalcomponent analysis or the like is performed for the spectroscopicreflectance so as to set three types of base vectors O1(λ), O2(λ), andO3(λ). Here, λ is n discrete values that are set at regular intervals ina visible light wavelength region (380 nm to 780 nm).

The weighting factors o1, o2, and o3 for each base vector in a casewhere the spectroscopic reflectance O(λ) of the body tissues isapproximated by applying the weighting factors to the base vectorsO1(λ), O2(λ), and O3(λ) can be calculated by using the followingEquation (8) based on the pixel values Pr, Pg, and Pb of the componentsR, G, and B of each pixel of the area (here, an area other than theclassification target area) as an estimation target, spectroscopicenergy I(λ) of illumination light, and the spectroscopic sensitivitiesSr(λ), Sg(λ), and Sb(λ) of imaging devices that configure the imagingsystem.

$\begin{matrix}{{\begin{bmatrix}{o\; 1} \\{o\; 2} \\{o\; 3}\end{bmatrix} = {\lbrack {{SI} \times O} \rbrack^{- 1} \times \begin{bmatrix}\Pr \\{Pg} \\{Pb}\end{bmatrix}}}{{where},{{SI} = \begin{bmatrix}{{{Sr}( {\lambda \; 1} )} \times {I( {\lambda \; 1} )}} & {{{Sr}( {\lambda \; 2} )} \times {I( {\lambda \; 2} )}} & \cdots & {{{Sr}( {\lambda \; n} )} \times {I( {\lambda \; n} )}} \\{{{Sg}( {\lambda \; 1} )} \times {I( {\lambda \; 1} )}} & {{{Sg}( {\lambda \; 2} )} \times {I( {\lambda \; 2} )}} & \cdots & {{{Sg}( {\lambda \; n} )} \times {I( {\lambda \; n} )}} \\{{{Sb}( {\lambda \; 1} )} \times {I( {\lambda \; 1} )}} & {{{Sb}( {\lambda \; 2} )} \times {I( {\lambda \; 2} )}} & \cdots & {{{Sb}( {\lambda \; n} )} \times {I( {\lambda \; n} )}}\end{bmatrix}}}{O = \begin{bmatrix}{O\; 1( {\lambda \; 1} )} & {O\; 2( {\lambda \; 1} )} & {O\; 3( {\lambda \; 1} )} \\{O\; 1( {\lambda \; 2} )} & {O\; 2( {\lambda \; 2} )} & {O\; 3( {\lambda \; 2} )} \\\vdots & \vdots & \vdots \\{O\; 1( {\lambda \; n} )} & {O\; 2( {\lambda \; n} )} & {O\; 3( {\lambda \; n} )}\end{bmatrix}}} & (8)\end{matrix}$

Actually, in Step d11, the weighing factors o1, o2, o3 are calculatedfor each pixel of an area other than the classification target area byusing Equation (8) described above. Thereafter, the base vectors O1(λ),O2(λ), and O3(λ) are weighted for each pixel using the calculatedweighting factors o1, o2, and o3, and the spectroscopic reflectance O(λ)at each pixel of the area other than the classification target area isestimated.

Next, the specific-wavelength component calculating unit 233 acalculates the pixel value of a specific wavelength component (awavelength component that cannot be easily absorbed or scattered insidethe body) based on the spectroscopic reflectance estimated in Step d11,and subsequently, the pixel-value calculating unit 23 a sets thecalculated pixel value of the specific wavelength component as athree-dimensional pixel value of the area other than the classificationtarget area (Step d13). In a case where the spectroscopic sensitivity ofthe imaging device is limited to a wavelength component that cannot beeasily absorbed or scattered inside the body, a pixel value thatcorresponds to the three-dimension image of the body tissue with highaccuracy can be acquired. The absorption of light inside the body mainlyoccurs in the absorption band of the blood (absorption bands ofoxy-hemoglobin and dioxy-hemoglobin contained in the blood), and thescattering of light can easily occur for a short wavelength component.Thus, according to the second embodiment, the wavelength component thatcannot be easily absorbed or scattered inside the body is set to a bandother than the absorption band of the blood and a long wavelengthcomponent that is a predetermined threshold value or more. Then, thiswavelength component is set as the specific wavelength component, andthe pixel value Px is calculated by using the following Equation (9)based on the spectroscopic sensitivity Sx(λ) of an imaging deviceassumed to have sensitivity for the specific wavelength component, thespectroscopic energy I(λ) of the illumination light, and the estimatedspectroscopic reflectance O(λ).

$\begin{matrix}{{{Px}\begin{bmatrix}{{{Sx}( {\lambda \; 1} )} \times {I({\lambda 1})}} & {{{Sx}( {\lambda \; 2} )} \times {I({\lambda 2})}} & \cdots & {{{Sx}( {\lambda \; n} )} \times {I( {\lambda \; n} )}}\end{bmatrix}} \times \begin{bmatrix}{O( {\lambda \; 1} )} \\{O( {\lambda \; 2} )} \\\vdots \\{O( {\lambda \; n} )}\end{bmatrix}} & (9)\end{matrix}$

In addition, in this example, the spectroscopic information is estimatedby using three types of base vectors and the components R, G, and B ofeach pixel. On the other hand, in a case where the present invention isapplied to an endoscope that can image, for example, three or morewavelength components, the spectroscopic information may be estimatedbased on the acquired wavelength components and base vectors of a typecorresponding to the number of the wavelength components.

Next, the pixel-value complementing unit 24 a complements the pixelvalues of the areas of the blood vessel, the mirror reflection portion,and the color-changed lesion as the classification target areas byperforming a pixel-value complementing process (Step d15). According tothe second embodiment, the pixel-value complementing process isperformed by using function approximation. Through this process, thethree-dimensional pixel values of the classification target area arecomplemented based on the pixel values of the specific wavelengthcomponent that are calculated as the three-dimensional pixel values ofthe area (particularly, the area of the periphery of the classificationtarget area) other than the classification target area in the formerstage of Step d13.

FIG. 17 is a conceptual diagram illustrating pixel-value complementingusing function approximation. In FIG. 17, while the three-dimensionalpixel values calculated in the former stage of Step d13, which isillustrated in FIG. 12, in advance are denoted by solid lines on theperiphery of the area, the pixel values of the intraluminal image aredenoted by dotted lines on the inside of the area that are complementedbased on the three-dimensional pixel values of the periphery of thearea. In the pixel-value complementing process according to the secondembodiment, as illustrated in FIG. 17( a), a plurality of sample pixelsP51 is selected from the inside of at least two periphery areas opposingeach other with the area interposed therebetween. The reason for this isfor preventing the complementation process from being influenced by thechange in the pixel value in the boundary portion of the complementationtarget area, and periphery pixels that are apart from the pixel of theboundary portion of the area by a predetermined distance or more areselected as the sample pixels P51. Then, the function formula of theapproximated curve is acquired based on the pixel values of theplurality of sample pixels P51 that have been selected, and thethree-dimensional pixel values of the inside of the area are acquired bycomplementing the pixel values of the inside of the area based on theacquired function formula of the approximated curve as denoted by asolid line in FIG. 17( b).

FIG. 18 is a flowchart illustrating the detailed processing sequence ofthe pixel-value complementing process according to the secondembodiment. In the pixel-value complementing process, as illustrated inFIG. 18, first, the pixel-value complementing unit 24 a sets aclassification target area (complementation target area) as a processingtarget (Step f1). Next, the sample pixel selecting unit 244 a performsan expansion process for the complementation target area and sets thesample pixels (Step f3). Described in more detail, it may be set suchthat the remaining pixels are selected based on a difference between aresult of performing a known expansion process (reference: “DigitalImage Processing”, CG-ARTS Society, 179P, Expansion Process) N₁ times(here, N₁ is a predetermined value) and a result of performing the knownexpansion process N₂ times (here, N₂ is a predetermined value and is N₁or less).

Next, the function approximation unit 243 a acquires the functionformula of the approximated curve based on the pixel values of thesample pixels (Step f5). In this example, as the function formula of thecurved surface, a quadratic function represented in the followingEquation (10) is used. Here, x and y are the coordinates of a pixel, andz is a pixel value.

z=ax ² +by ² +cxy+dx+ey+f  (10)

The coefficients a to f of the function formula represented in Equation(10) are acquired by using the following Equation (11) acquired by usinga least-square method based on the coordinates (xi, yi) (i=1 to n; n isthe number of sample pixels) of the sample pixels and the pixel valuezi.

$\begin{matrix}{{\begin{bmatrix}a \\b \\c \\d \\e \\f\end{bmatrix} = {( {A^{t} \times A} )^{- 1} \times A^{t} \times \begin{bmatrix}{z\; 1} \\{z\; 2} \\\vdots \\{zn}\end{bmatrix}}}{{where},\text{}{A = \begin{bmatrix}{x\; 1^{2}} & {y\; 1^{2}} & {x\; 1y\; 1} & {x\; 1} & {y\; 1} & 1 \\{x\; 2^{2}} & {y\; 2^{2}} & {x\; 2y\; 2} & {x\; 2} & {y\; 2} & 1 \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\{x\; n^{2}} & {yn}^{2} & {xnyn} & {xn} & {yn} & 1\end{bmatrix}}}} & (11)\end{matrix}$

Then, the pixel-value complementing unit 24 a complements the pixelvalues based on the acquired function formula (Step f7). Described inmore detail, the pixel values are calculated by using Equation (10)described above for each pixel of the complementation target area basedon the coordinates thereof.

Thereafter, the pixel-value complementing unit 24 a determines whetheror not the pixel values of all the complementation target areas havebeen completed. In a case where all the complementation target pixelshave not been complemented (No in Step f9), the pixel-valuecomplementing unit 24 a sets the classification target area that has notbeen completed as a complementation target area (Step f11) and performsthe process of Steps f3 to f9 again. On the other hand, in a case wherethe pixel values of all the classification target areas are complemented(Yes in Step f9), the process is returned to Step d15 illustrated inFIG. 12 and then proceeds to Step d21.

In addition, in a case where it is determined that there is no extractedarea in Step d5 (No in Step d5) or in a case where it is determined thatthere is no area classified as the area of the blood vessel, the mirrorreflection portion, or the color-changed lesion in Step d9 (No in Stepd9), first, the spectroscopic information estimating unit 232 a of thepixel-value calculating unit 23 a estimates the spectroscopicreflectance of each pixel of the entire area of the intraluminal imagebased on the pixel values of the components R, G, and B of the entirearea of the intraluminal image (Step d17). Thereafter, thespecific-wavelength component calculating unit 233 a calculates thepixel values of the specific wavelength component (a wavelengthcomponent that cannot be easily absorbed or scattered inside the body)based on the spectroscopic reflectance estimated in Step d17, andsubsequently, the pixel-value calculating unit 23 a sets the calculatedpixel values of the specific wavelength component as thethree-dimensional pixel values of the pixels of the entire area of theintraluminal image (Step d19).

Finally, the calculation unit 20 a outputs the three-dimensional pixelvalues calculated in Steps d13 and d15 based on the pixel values of thepixels of the intraluminal image or the three-dimensional pixel-valueinformation that is set as the three-dimensional pixel values calculatedin Step d19 (Step d21), and the process of the image processingapparatus 10 a ends.

As described above, according to the second embodiment, a concave arearepresenting a pixel value less than the average value of the pixels ofthe periphery pixels and a convex area representing a pixel valuegreater than the average value of the pixel values of the peripherypixels are extracted as areas that are strongly influenced by lightabsorption, mirror reflection, or the like, more particularly,classification target candidate areas that are candidates for the areasof a blood vessel, a mirror reflection portion, or a color-changedlesion, based on the information of the change in the pixel value withrespect to the periphery pixels. Then, based on the steepness of thechange in the pixel value of the boundary portion of the classificationtarget candidate area, the areas of the blood vessel, the mirrorreflection portion, and the color-changed lesion that are stronglyinfluenced by light absorption, a mirror reflection portion, and thelike out of the classification target candidate areas are classified asthe classification target areas. Accordingly, areas of the intraluminalimage in which the pixel value does not correspond to thethree-dimensional shape of the imaging target can be specified.

In addition, according to the second embodiment, in an area other thanthe classification target area that are areas in which a specific pixelvalue does not correspond to the three-dimensional shape of the imagingtarget as described above, by acquiring the pixel values of a specificwavelength component that is specified in accordance with the degree ofabsorption or scattering inside the body, the three-dimensional pixelvalues corresponding to the three-dimensional shape of the body tissueare calculated. Then, function approximation is performed based on thepixel values of the specific wavelength component that are acquired, forexample, as the three-dimensional pixel values of the periphery of theclassification target area, and the pixel values of the areas of theblood, the mirror reflection portion, and the color-changed lesion asthe classification target areas are complemented, whereby thethree-dimensional pixel values of the inside of the classificationtarget area are acquired. Therefore, the three-dimensional pixel valuesthat appropriately represent the three-dimensional shape of the bodytissue are calculated for the entire area of the intraluminal image andcan be output as the three-dimensional pixel-value information.

Third Embodiment

First, the configuration of an image processing apparatus according to athird embodiment will be described. FIG. 19 is a schematic block diagramillustrating the main configuration of an image processing apparatus 10b according to the third embodiment. The same reference numeral isassigned to the same configuration as that described in the firstembodiment. The image processing apparatus 100 b according to the thirdembodiment includes a calculation unit 20 b and a recording unit 30 b,as illustrated in FIG. 19. This image processing apparatus 10 b, forexample, similarly to the first embodiment, is built in an endoscope andoutputs calculated three-dimensional pixel value information byreceiving an intraluminal image that is imaged by the endoscope as aninput and processing the intraluminal image.

This calculation unit 20 b includes the area extracting unit 21configured similarly to that of the first embodiment, an areaclassifying unit 22 b, the pixel-value calculating unit 23 that isconfigured similarly to that of the first embodiment, and thepixel-value complementing unit 24 that is configured similarly to thefirst embodiment, and only the configuration of the area classifyingunit 22 b is different from that of the first embodiment.

The area classifying unit 22 b according to the third embodimentclassifies areas of a blood vessel, a mirror reflection portion, and acolor-changed lesion out of the classification-target candidate areas asthe classification target areas based on the continuity of the change inthe pixel value (curved shape) of periphery portions of theclassification target candidate area that are represented by the pixelvalues of the periphery portions of the classification target candidateareas. This area classifying unit 22 b includes a periphery areafunction approximation unit 226 b and a periphery continuity determiningunit 227 b. The periphery area function approximation unit 226 bperforms function approximation of the curved shape of the peripheryportions of at least two classification target candidate areas opposingeach other with the classification target candidate area interposedtherebetween. The periphery continuity determining unit 227 b determinesthe continuity of the change in the pixel value between the peripheryportions of the area by comparing results of function approximation ofthe periphery portions of at least two classification target candidateareas.

In the recording unit 30 b, an image processing program 31 b used forcalculating three-dimensional pixel value information from theintraluminal image is recorded.

Next, the detailed processing sequence performed by the image processingapparatus 10 b according to the third embodiment will be described. FIG.20 is an entire flowchart illustrating the processing sequence performedby the image processing apparatus 10 b according to the thirdembodiment. The process described here is realized by executing theimage processing program 31 b recorded in the recording unit 30 b byusing the calculation unit 20 b. In FIG. 20, the same reference numeralis assigned to the same processing sequence as that of the firstembodiment.

According to the third embodiment, in a case where the calculation unit20 b determines that there is an area extracted as an area of a specificcolor (classification target candidate area) in Step a5 (Yes in Stepa5), the area classifying unit 22 b classifies areas of a blood vessel,a mirror reflection portion, and a color-changed lesion, which are theclassification target areas, out of the classification-target candidateareas based on the continuity of the change in the pixel value of theperiphery portions of the classification target candidate area byperforming an area classifying process (Step g7). As described in thefirst embodiment, there is a possibility that a difference in level of amucous membrane, a mucous membrane groove, or the like other than theblood vessel, the mirror reflection portion, and the color-changedlesion is included in the area of the specific color that is extractedbased on the color feature data in Step a3. Thus, in the areaclassifying process according to the third embodiment, both areas areclassified by determining the continuity of the change in the pixelvalue of the periphery portions of the area based on the curved shape ofthe periphery portions of the classification-target candidate area thatis represented by the pixel values of the periphery portions of theclassification-target candidate area.

In addition, before the classification, pixels having the same pixelvalue within the candidate area image are connected to each otherthrough a labeling process, which is similar to that of the firstembodiment.

Here, before the detailed processing sequence of the area classifyingprocess is described, the features of the area periphery portions ofeach category area of a blood vessel, a mirror reflection portion, acolor-changed lesion, a difference in level of a mucous membrane, and amucous membrane groove included in the area of the specific color willbe described. FIG. 21A is a schematic diagram illustrating a change inthe pixel value in the area periphery portion of the mirror reflectionportion 3 along cross-section direction a illustrated in FIG. 5 referredto in the first embodiment. FIG. 21B is a schematic diagram illustratinga change in the pixel value of the area periphery portion alongcross-section direction b in a case where the color-changed lesion 5illustrated in FIG. 5 is a white lesion. FIG. 21C is a schematic diagramillustrating a change in the pixel value in the area periphery portionof the blood vessel 2 along the cross-section direction c illustrated inFIG. 5. FIG. 21D is a schematic diagram illustrating a change in thepixel value of the area periphery portion along cross-section directiond in a case where the color-changed lesion 5 illustrated in FIG. 5 is ared or black lesion. FIG. 21E is a schematic diagram illustrating achange in the pixel value of the area periphery portion alongcross-section direction e in a case where the mucous membrane undulation7 illustrated in FIG. 5 is a difference in level in a mucous membrane.FIG. 21F is a schematic diagram illustrating a change in the pixel valueof the area periphery portion along cross-section direction f in a casewhere the mucous membrane undulation 7 illustrated in FIG. 5 is a mucousmembrane groove. In FIGS. 21A, 21B, 21C, 21D, 21E, and 21F, approximatedcurved faces that approximate the curved shapes of two area peripheryportions, which oppose each other, of a corresponding category area aredenoted by dashed-two dotted lines.

As described in the first embodiment, the pixel values of theintraluminal image are composed of values of the components R, G, and B.Although there are differences in the scale between the values, thetendencies of the changes are similar to each other. Here, the mirrorreflection portion 3 along cross-section direction a illustrated in FIG.5, a color-changed lesion 5 such as a white lesion, a red lesion, or ablack lesion along cross-section directions b and d, and a blood vessel2 along cross-section direction c are continuous in the shapes, asillustrated in FIGS. 21A, 21B, 21C, 21D and have continuity in thechange in the pixel value between area periphery portions. Thus, whenthe function formulas of the approximated curve faces acquired byapproximate the curved shapes of the area periphery portions, which arerepresented by the pixel values of the area periphery portions, that arecontinuous as above are acquired, a difference value of the pixel valuesat the same coordinates is small. On the other hand, since thedifference in level of the mucous membrane undulation 7 such as adifference in level of a mucous membrane or a mucous membrane groovealong cross-section directions e and f illustrated in FIG. 5 is notcontinuous in the shape, there is no continuity in the change in thepixel value between the area periphery portions, as illustrated in FIGS.21E and 21F. When the function formulas of the approximated curved facesacquired by approximate the curved shapes of the periphery portions thatare represented by the pixels values of the area periphery portions thatdo not have continuity are acquired, a difference between pixel valuesat the same coordinates is large. In the area classifying processaccording to the third embodiment, the areas of a blood, a mirrorreflection portion, and a color-changed area, which are theclassification target areas, out of the classification target candidateareas are classified based on such features.

FIG. 22 is a flowchart illustrating the detailed processing sequence ofthe area classifying process according to the third embodiment. In thearea classifying process, as illustrated in FIG. 22, first, the areaclassifying unit 22 b sets a classification-target candidate area(processing candidate area) as a processing target (Step h1).

Next, the periphery area function approximation unit 226 b sets two areaperiphery portions opposing each other with a principal axis of the areainterposed therebetween (Step h3). This can be set by dividing theremaining pixels based on a difference between a result of performing aknown expansion process N₁ times (here, N₁ is a predetermined value) forthe process candidate areas and a result of performing the expansionprocess N₂ times (here, N₂ is a predetermined value that is N₁ or less)into two with a principal axis (reference: “Digital Image Processing”,CG-ARTS Society, 183P, Principal Axis) as shape feature data of thearea. Then, function formulas of two approximated curved faces areacquired based on the pixel values of the inside of the area peripheryarea (Step h5). The method of acquiring the function formula of theapproximated curved face is the same as that of the second embodimentthat has been described in Step f5 illustrated in FIG. 18.

Next, the periphery continuity determining unit 227 b determines whetherthe change in the pixel value of the two area periphery portions iscontinuous. As the detailed processing sequence, first, the pixel valuesof the inside of the processing candidate area are calculated for eachfunction formula of two approximated curved faces by using the samemethod as that described in Step f7 illustrated in FIG. 18 in the secondembodiment. Next, a sum of absolute values of the pixel values at thesame coordinates inside the processing candidate areas calculated foreach function formula is acquired. Thereafter, by determining whether ornot the acquired value is a predetermined threshold value or less, it isdetermined whether or not the change in the pixel value between the areaperiphery portions is continuous. Then, in a case where the change inthe pixel value between the two area periphery portions is determinednot to be continuous (No in Step h7), the processing candidate area isclassified as a difference in level of a mucous membrane or a mucousmembrane groove (Step h9). On the other hand, in a case where the changein the pixel value is determined to be continuous (Yes in Step h7), theprocessing candidate area is classified as a blood vessel, a mirrorreflection portion, or a color-changed lesion (Step h11).

Thereafter, the area classifying unit 22 b determines whether all theclassification-target candidate areas have been classified. In a casewhere all the classification-target candidate areas have not beenclassified (No in Step h13), the area classifying unit 22 b sets aclassification-target candidate area that has not been classified as theprocessing candidate area (Step h15) and performs the process of Stepsh3 to h13 again. On the other hand, in a case where all theclassification-target candidate areas have been classified (Yes in Steph13), the process is returned to Step g7 illustrated in FIG. 20 andproceeds to Step a9. Consequently, a classification-target area image isgenerated in which positive values (label values) used for identifyingareas are set to the pixels of the areas classified as the blood vessel,the mirror reflection portion, and the color-changed lesion as theclassification target areas, and “0” is set to pixels of areas otherthan the classification target areas.

As described above, according to the third embodiment, the areas of theblood, the mirror reflection portion, and the color-changed lesion thatare strongly influenced by light absorption or mirror reflection areclassified out of the classification-target candidate areas as theclassification target areas based on the continuity of the curved shapethat is represented by the pixel values of the periphery portions of theclassification target candidate areas. Accordingly, the same advantagesas those of the first embodiment or the second embodiment can beacquired and thus it is possible to specify an area in which the pixelvalues do not correspond to the three-dimensional image from theintraluminal image. Then, three-dimensional pixel values thatappropriately represent a three-dimensional shape of the body tissue inthe entire area of the intraluminal image can be calculated and beoutput as the three-dimensional pixel value information.

Furthermore, the image processing apparatus 10 according to the firstembodiment, the image processing apparatus 10 a according to the secondembodiment, and the image processing apparatus 10 b according to thethird embodiment described above can be realized by executing a programprepared in advance by using a computer system such as a personalcomputer or a workstation. Hereinafter, a computer system that has thesame functions as those of the image processing apparatuses 10, 10 a,and 10 b described in the first to third embodiments and executes theimage processing programs 31, 31 a, and 31 b will be described.

FIG. 23 is a system configuration diagram illustrating the configurationof a computer system 400 according to this modified example, and FIG. 24is a block diagram illustrating the configuration of a main body unit410 that configures the computer system 400. As illustrated in FIG. 23,the computer system 400 includes: the main body unit 410; a display 420that is used for displaying information such as an image on a displayscreen 421 in accordance with an instruction transmitted from the mainbody unit 410; a keyboard 430 that is used for inputting various typesof information to the computer system 400; and a mouse 440 that is usedfor designating an arbitrary position on the display screen 421 of thedisplay 420.

In addition, the main body unit 410 of this computer system 400, asillustrated in FIGS. 23 and 24, includes: a CPU 411; a RAM 412; a ROM413; a hard disk drive (HDD) 414; a CD-ROM drive 415 that accepts aCD-ROM 460; a USB port 416 to which a USB memory 470 can be detachablyconnected; an I/O interface 417 that connects the display 420, thekeyboard 430, and the mouse 440 together; and a LAN interface 418 thatis used for being connected to a local area network or a wide areanetwork (LAN/WAN) N1.

Furthermore, to this computer system 400, a modem 450 that is used forbeing connected to a public circuit N3 such as the Internet and apersonal computer (PC) 481 as another computer system, a server 482, aprinter 483, and the like are connected through the LAN interface 418and the local network or the wide area network N1.

This computer system 400 realizes the image processing apparatus (forexample, the image processing apparatus 10 according to the firstembodiment, the image processing apparatus 10 a according to the secondembodiment, or the image processing apparatus 10 b according to thethird embodiment) by reading out and executing an image processingprogram (for example, the image processing program 31 according to thefirst embodiment, the image processing program 31 a according to thesecond embodiment, or the image processing program 31 b according to thethird embodiment) stored on a recording medium. Here, the recordingmedia includes all types of recording media on which an image processingprogram is recorded so as to be readable by using the computer system400 such as “portable-type physical media” including an MO disc, a DVDdisc, a flexible disc, (FD), an IC card, and the like in addition to theCD-ROM 460 and the USB memory 470, “fixed-type physical media” includingthe HDD 414, the RAM 412, the ROM 413, and the like that can beinternally or externally included in the computer system 400, and“communication media” such as a public circuit N3 that is connectedthrough the modem 450, a local area network or a wide area network N1 towhich the PC 481 as another computer system or the server 482 isconnected, and the like that store a program for a short time when theprogram is transmitted.

In other words, the image processing program is recorded on a recordingmedium such as a “portable-type physical medium”, a “fixed-type physicalmedium”, or a “communication medium” in a computer-readable form, andthe image processing apparatus is realized by reading out the imageprocessing program from such a recording medium and executing the imageprocessing program by using the computer system 400. In addition, theimage processing program is not limited as being executed by thecomputer system 400, and the present invention can be similarly appliedto a case where the PC 481 as another computer system or the sever 482executes the image processing program or a case where the PC 481 and thesever 482 cooperatively execute the image processing program.

In addition, in each embodiment described above, the image processingapparatus that is built in an endoscope and processes an intraluminalimage that is imaged by the endoscope has been described. However, theintraluminal image as a processing target of the image processingapparatus according to the present invention is not limited to an imagethat is captured by the endoscope. For example, recently, an eating-typeendoscope (capsule endoscope) in which an imaging device, acommunication device that transmits image data imaged by the imagingdevice outside the body in a wireless manner, and the like are includedinside a capsule-type casing is developed, and the present invention canbe similarly applied even in a case where the intraluminal image imagedby the capsule endoscope is processed.

In addition, the present invention is not limited to each of the firstto third embodiments described above, and various inventions can beconfigured by appropriately combining a plurality of constituentelements disclosed in the first to third embodiments and the modifiedexamples. For example, a configuration may be employed in which severalconstituent elements are excluded from all the constituent elementsillustrated in each embodiment. Alternatively, the constituent elementsillustrated in different embodiments may be appropriately combined. As aconcrete example, an image processing apparatus may be configured byappropriately selecting and combining the area extracting unit 21, thearea classifying unit 22, the pixel-value calculating unit 23, and thepixel-value complementing unit 24 according to the first embodiment, thearea extracting unit 21 a, the area classifying unit 22 a, thepixel-value calculating unit 23 a, and the pixel-value complementingunit 24 a according to the second embodiment, and the area classifyingunit 22 b according to the third embodiment, as the area extractingunit, the area classifying unit, the pixel-value calculating unit, andthe pixel-value complementing unit.

According to the present invention described above, an area of anintraluminal image in which the pixel value does not correspond to athree-dimensional shape can be specified.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. An image processing apparatus comprising: an area extracting unitthat extracts a candidate area of a classification target area in whicha pixel value does not correspond to a three-dimensional shape of animaging target based on pixel values of an intraluminal image acquiredby imaging the inside of a lumen or information of a change in pixelvalues of peripheral pixels; and an area classifying unit thatclassifies the classification target area out of the candidate areabased on the pixel values of the inside of the candidate area, aboundary portion of the candidate area, or a periphery portion of thecandidate area.
 2. The image processing apparatus according to claim 1,wherein the area classifying unit classifies the classification targetarea out of the candidate area based on unevenness information andsmoothness of a curved shape of the candidate area that is representedby the pixel values of the inside of the candidate area.
 3. The imageprocessing apparatus according to claim 1, wherein the area classifyingunit classifies the classification target area out of the candidate areabased on steepness of the change in the pixel values of the boundaryportion of the candidate area.
 4. The image processing apparatusaccording to claim 1, wherein the area classifying unit classifies theclassification target area out of the candidate area based on continuityof a curved shape that is represented by the pixel values of theperiphery portion of the candidate area.
 5. The image processingapparatus according to claim 1, further comprising a pixel-valuecomplementing unit that complements the pixel values of theclassification target area based on the pixel values of an area otherthan the classification target area.
 6. The image processing apparatusaccording to claim 1, further comprising a pixel-value calculating unitthat calculates a pixel value of a specific-wavelength component that isspecified in accordance with the degree of absorption or scatteringinside a body, in an area other than the classification target area. 7.The image processing apparatus according to claim 5, further comprisinga pixel-value calculating unit that calculates a pixel value of aspecific wavelength component that is specified in accordance with thedegree of absorption or scattering inside a body, in an area other thanthe classification target area, wherein the pixel-value complementingunit complements the pixel values of the classification target areabased on the pixel values of the specific wavelength component that arecalculated by the pixel-value calculating unit.
 8. The image processingapparatus according to claim 1, wherein the area extracting unitincludes a specific-color area extracting unit that extracts aspecific-color area representing pixel values of a specific color as thecandidate area.
 9. The image processing apparatus according to claim 1,wherein the area extracting unit includes a concave area/convex areaextracting unit that extracts a concave area representing a pixel valueless than an average value of the pixel values of periphery pixels and aconvex area representing a pixel value greater than the average value ofthe pixel values of the periphery pixels as the candidate areas.
 10. Theimage processing apparatus according to claim 9, wherein the concavearea/convex area extracting unit includes: a directional differencecalculating unit that calculates difference values between a pixel ofinterest and an average value of the periphery pixels opposing eachother in a predetermined direction with the pixel of interest beinglocated at the center thereof for a plurality of directions; a maximumvalue/minimum value calculating unit that calculates a maximum value anda minimum value of the difference values for the plurality ofdirections; and a threshold value processing unit that performsthreshold-value processing for the maximum value and the minimum valueof the difference values; wherein the concave area and the convex areaare extracted based on a result of the threshold-value processing of thethreshold value processing unit.
 11. The image processing apparatusaccording to claim 2, wherein the area classifying unit includes: asecond derivative calculating unit that calculates a second derivativevalue by applying a second derivative filter to the pixels of the insideof the candidate area that is extracted by the area extracting unit; aninternal unevenness determining unit that determines unevenness of thecurved shape of the candidate area that is represented by the pixelvalues of the inside of the candidate area based on a sign of the secondderivative value of the pixels of the inside of the candidate area; andan internal smoothness determining unit that determines smoothness ofthe curved shape of the candidate area that is represented by the pixelvalues of the inside based on an absolute value of the second derivativevalue of the pixels of the inside of the candidate, wherein theclassification target area is classified based on determination resultsof the internal unevenness determining unit and the internal smoothnessdetermining unit.
 12. The image processing apparatus according to claim3, wherein the area classifying unit includes: a first derivativecalculating unit that calculates a first derivative value by applying afirst derivative filter to the pixels of the boundary portion of thecandidate area; and a boundary steepness determining unit thatdetermines steepness of the change in the pixel value of the boundaryportion of the candidate area based on the first derivative value of thepixels of the boundary portion of the candidate area, wherein theclassification target area is classified based on a determination resultof the boundary steepness determining unit.
 13. The image processingapparatus according to claim 4, wherein the area classifying unitincludes: a periphery area function approximation unit that performsfunction approximation of the curved shape of the candidate area that isrepresented by the pixel values of at least two area periphery portionsopposing each other with the candidate area interposed therebetween; anda periphery continuity determining unit that determines continuity ofthe two area periphery portions by comparing results of the functionapproximation of the two area periphery portions, wherein theclassification target area is classified based on a determination resultof the periphery continuity determining unit.
 14. The image processingapparatus according to claim 6, wherein the intraluminal image iscomposed of a plurality of wavelength components, and wherein thepixel-value calculating unit includes a specific-wavelength componentselecting unit that selects the specific-wavelength component out of theplurality of wavelength components or wavelength components that areacquired by converting the plurality of wavelength components.
 15. Theimage processing apparatus according to claim 6, wherein theintraluminal image includes a plurality of wavelength components, andwherein the pixel-value calculating unit includes: a spectroscopicinformation estimating unit that estimates spectroscopic information ofa body tissue based on the plurality of wavelength components; and aspecific-wavelength component calculating unit that calculates a pixelvalue of the specific-wavelength component based on the spectroscopicinformation.
 16. The image processing apparatus according to claim 5,wherein the pixel-value complementing unit includes a morphologyprocessing unit that performs morphology processing based on the pixelvalues of the periphery of the classification target area, and whereinthe pixel values of the classification target area are complementedbased on a result of the morphology processing of the morphologyprocessing unit.
 17. The image processing apparatus according to claim16, wherein the morphology processing unit includes a structure elementgenerating unit that generates a structure element of the morphologyprocessing based on feature data of the classification target area, andthe morphology processing is performed by using the structure elementgenerated by the structure element generating unit.
 18. The imageprocessing apparatus according to claim 5, wherein the pixel-valuecomplementing unit includes a function approximation unit that performsfunction approximation based on the pixel values of the periphery of theclassification target area, and the pixel values of the classificationtarget area are complemented based on a result of the functionapproximation of the function approximation unit.
 19. The imageprocessing apparatus according to claim 18, wherein the functionapproximation unit includes a sample-pixel selecting unit that selectsat least one sample pixel that is used for the function approximationfrom each of at least two periphery areas opposing each other with theclassification target area interposed therebetween, and the functionapproximation is performed based on the sample pixel.
 20. A method ofprocessing an image, the method comprising: extracting a candidate areaof a classification target area in which a pixel value does notcorrespond to a three-dimensional shape of an imaging target based onpixel values of an intraluminal image acquired by imaging the inside ofa lumen or information of a change in pixel values of peripheral pixels;and classifying the classification target area out of the candidate areabased on the pixel values of the inside of the candidate area, aboundary portion of the candidate area, or a periphery portion of thecandidate area.
 21. A non-transitory computer-readable recording mediumwith an executable program stored thereon, wherein the program instructsa processor to perform: extracting a candidate area of a classificationtarget area in which a pixel value does not correspond to athree-dimensional shape of an imaging target based on pixel values of anintraluminal image acquired by imaging the inside of a lumen orinformation of a change in pixel values of peripheral pixels; andclassifying the classification target area out of the candidate areabased on the pixel values of the inside of the candidate area, aboundary portion of the candidate area, or a periphery portion of thecandidate area.